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Abstract 

Background: Patients and clinicians expect patient decision aids to be based on the best available research 
evidence. Since 2005, this expectation has translated into a quality dimension of the International Patient Decision 
Aid Standards. 

Methods: We reviewed the 2005 standards and the available literature on the evidence base of decision aids as 
well as searched for parallel activities in which evidence is brought to bear to inform clinical decisions. In 
conducting this work, we noted emerging and research issues that require attention and may inform this quality 
dimension in the future. 

Results: This dimension requires patient decision aids to be based on research evidence about the relevant options 
and the nature and likelihood of their effect on outcomes that matter to patients. The synthesis of evidence should be 
comprehensive and up-to-date, and the evidence itself subject to critical appraisal. Ethical (informed patient choice), 
quality-of-care (patient-centered care), and scientific (evidence-based medicine) arguments justify this requirement. 
Empirical evidence suggests that over two thirds of available decision aids are based on high-quality evidence 
syntheses. Emerging issues identified include the duties of developers regarding the conduct of systematic reviews, the 
impact of comparative effectiveness research, their link with guidelines based on the same evidence, and how to 
present the developers' confidence in the estimates to the end-users. Systematic application of the GRADE system, 
common in contemporary practice guideline development, could enhance satisfaction of this dimension. 

Conclusions: While theoretical and practical issues remained to be addressed, high-quality patient decision aids 
should adhere to this dimension requiring they be based on comprehensive and up-to-date summaries of critically 
appraised evidence. 



Background 

The original 2005 International Patient Decision Aid 
Standards (IPDAS) formulation included a key quality 
dimension then labeled as "basing information on up- 
to-date scientific evidence". This dimension reflected the 
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Standards panel's belief that these tools should advance 
evidence-based practice. Since 2005, much work has 
taken place that advances the use of research evidence 
to develop tools that support clinical decision making, 
requiring a review and update in the formulation of this 
quality dimension. For this iteration, we put forward a 
more detailed description of this dimension. We now 
require patient decision aids to base their information 
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on comprehensive, critically appraised, and up-to-date 
syntheses of the scientific evidence. 

In this manuscript, we present a detailed description 
of this quality dimension, the theoretical and empirical 
justification for evaluating patient decision aids accord- 
ing to this dimension, and discuss the emerging issues 
and research areas for consideration and discussion 
identified during this endeavor. 

The dimension and its features 

IPDAS requires patient decision aids to base their infor- 
mation on comprehensive, critically appraised, and up-to- 
date syntheses of the scientific evidence. This refers to the 
information about the various relevant options, and about 
the descriptions and likelihoods of those options' effects 
on the outcomes of most importance to patients. 

The term "scientific evidence" refers to a body of empiri- 
cal observations about the options and their consequences. 
These observations should be made with some degree of 
protection against systematic and random error. 

By "comprehensive", we do not mean the scope and 
range of the information presented in the decision aid. 
Instead, in defining this quality dimension, "comprehen- 
sive" refers to the extent to which decision aid developers 
have thoroughly considered all the pertinent scientific 
evidence addressing each aspect that they chose to pre- 
sent in their decision aid. 

The requirement that the evidence be "critically 
appraised" means that decision aid developers will use 
evidence derived from systematic reviews that: a) avoid 
selection bias (i.e., avoid introducing bias into the review 
in the selection of studies to include); b) carefully and 
reproducibly assess the quality of the incorporated 
reviewed studies (i.e., the studies' protection from error 
and bias); c) summarize the estimated pertinent effects 
(ideally quantitatively in a meta-analysis); d) indicate the 
extent to which these estimates are trustworthy; and e) 
assess the extent to which selective reporting and publi- 
cation bias may corrupt the body of evidence. Develo- 
pers must present this synthesized evidence in the 
decision aid itself in a way such that it a) conveys that 
the aid is offering the "best available" synthesized infor- 
mation, and b) indicates— using symbols, numbers, or 
phrases— the degree of confidence attributed to that 
information, given the quality of the scientific evidence 
upon which it is based. 

The dates when the relevant systematic reviews were 
searched for, compiled, critically appraised, and synthe- 
sized by decision aid developers should be reported and 
should be sufficiently recent given the pace of progress in 
the particular field to be considered "up-to-date". This 
implies that decision aid developers should develop a 
sense of the speed with which evidence that matters 
accrues in their area of work, and implement an update 



policy. Developers should report a version date, and, 
when pertinent, a "better by" or "expiration date" to com- 
municate to decision aid users about the speed of evi- 
dence accrual in the field and the optimal timing of 
future updates. 

Theoretical justification for evaluating patient 
decision aids on this quality dimension 

As we described above, clinicians and patients expect 
patient decision aids to be evidence-based tools. This 
alone could justify this dimension (and indeed it did in 
its 2005 formulation). Here, we push the rationale further 
a) by connecting it with three fundamental arguments - 
an ethical argument, a quality-of-care argument, and a 
scientific argument - that are central to contemporary 
clinical care, and b) by describing recent advances in the 
rating of the degree of confidence in the estimates of 
effect justified by the state of the available scientific 
evidence. 

Supporting arguments 

Three fundamental arguments central to contemporary 
clinical care justify the IPDAS's evidence dimension. 
Informed Patient Choice 

One key principle driving the development of patient deci- 
sion aids is the ethical argument for informing patients 
about their health care choices. Respect for patient auton- 
omy is a governing principle of medical ethics [1], and is 
generally understood to refer to an individual's ability to 
make and carry out informed health care decisions based 
on unbiased and thoughtful deliberation. The American 
Board of Internal Medicine, the American College of 
Physicians, and the European Federation of Internal Medi- 
cine state in their charter on medical professionalism that 
"physicians must be honest with their patients. ..and 
ensure that patients are completely and honestly informed 
before the patient has consented to treatment and after 
treatment has occurred." [2] To be "completely and hon- 
estly informed" so that autonomy can be exercised 
requires access to unbiased information that is based on a 
high-quality synthesis of the available evidence that is rele- 
vant to the patient's clinical situation and that acknowl- 
edges where uncertainty exists because of the quality or 
quantity of that evidence. When used as part of a shared 
decision making process, high-quality patient decision aids 
that are based on the best available clinical evidence sup- 
port clinicians in fulfilling their ethical obligation to pro- 
mote autonomy by ensuring that patients are fully 
informed about their health care choices. 
Patient-Centered Care 

The growing emphasis on patient-centered care is 
another driving force behind the development of patient 
decision aids. Principles of patient-centered care require 
that patients actively participate in decision making and 
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be provided with the information and support they need 
to mal<e informed choices. Work by the Picker Institute 
and others has identified respect for the patient's values 
and preferences, as well as access to clear, high-quality 
information and education to be among the important 
characteristics of patient-centered care [3]. Patient deci- 
sion aids support the practice of patient-centered care 
by ensuring that patients' preferences are informed and 
based on accurate expectations; this requires, in turn, 
that the tools be based in high-quality evidence that, 
when possible, is relevant to patients' individual risk 
profiles. 

Evidence-Based Medicine 

Over the last 20 years, evidence-based medicine (EBM) 
has strongly influenced the practice of medicine [4] as it 
has required use of the best available evidence alongside 
clinical expertise to formulate recommendations to 
patients that are pertinent to their context and sensitive 
to their values and preferences. To this extent, EBM fol- 
lows two principles. 

The first principle recognizes that not all observations 
and experiments are similarly protected from random 
and systematic error. The degree of protection from 
error offers confidence in the estimates of effect. There- 
fore, in the interests of fostering EBM, the information 
presented in patients' decision aids should include not 
only evidence-based estimates of the effects of the var- 
ious relevant options, but also an indication of the 
extent to which this evidence is protected from error 
(i.e., we can have confidence in the estimates of effect). 

The second principle holds that the evidence alone is 
never sufficient to fully inform a clinical decision. The 
evidence will be considered during the transactions 
between the treating clinician with a particular level of 
expertise and the patient with unique goals, values, and 
preferences; furthermore, the evidence will be applied in 
a particular biological, psychological, and sociocultural 
context. Therefore, in the interests of fostering EBM, 
the information presented in patients' decision aids 
needs to be directly applicable to the patients and prac- 
titioners using the decision aid, and tailored to indivi- 
dual patients' characteristics. 

Ethical (informed patient choice), quality (patient- 
centered care), and scientific (evidence-based medi- 
cine) justifications make it imperative that decision 
aids be based on comprehensive, critically appraised, 
and up-to-date scientific evidence. 

Rating the degree of confidence in the estimates of effect 

Decision aid developers may want to include informa- 
tion in their tool that is based on different forms of evi- 
dence. Natural history and prognostic information, for 
instance, often requires the developer to use large and 



long observational studies. Adverse effects, particularly 
those that are rare, may be better characterized in case 
reports, and their linkage to exposures ascertained 
through case-control studies. This notion of a hierarchy 
of evidence has received much attention and has 
resulted in an emerging consensus in the practice guide- 
lines movement about how to assess the confidence in 
the estimates of effect from the body of pertinent evi- 
dence: the GRADE approach. While we will now focus 
on this approach, we recognize that other approaches to 
grading the evidence exist, but they have substantial 
shortcomings that this method obviates. Furthermore, 
consistency in rating evidence from guideline to decision 
aid may facilitate the development of decision aids in 
conjunction with clinical practice guidelines efforts. 
The GRADE Approach 

GRADE (Grading of Recommendations Assessment, 
Development and Evaluation) is the most comprehen- 
sive approach developed for the purposes of formulating 
clinical practice guidelines [5]. It could be particularly 
helpful for decision aid developers, as an approach to 
explaining to patients the extent to which one can have 
confidence in the pertinent estimates of an option's 
effects. 

We will summarize the GRADE approach to grading 
evidence of effectiveness here, but developers should 
review the extensive published guidance (http://www. 
gradeworkinggroup.org). 

A key feature of this approach is that the assessment 
of quality applies to the body of evidence (not just to 
the individual study). This includes: a) the likelihood of 
bias (from the absence of protective features such as 
concealed randomization, blinding of pertinent groups, 
and analysis of participants as randomized) [6]; b) the 
likelihood of reporting or publication bias [7]; c) incon- 
sistency in results across studies [8]; d) any imprecision 
in the estimates of effect (e.g., wide confidence intervals) 
[9]; and d) the degree of indirectness, in which the 
results do not directly apply to the pertinent patients, 
the comparisons are inadequate, or the studies measure 
a surrogate of limited validity [10]. 

Randomized trials often provide high confidence in 
effect estimates. The task is to identify limitations that 
would reduce our confidence in those estimates. Obser- 
vational studies often provide low confidence in effect 
estimates, limitations further reduce that confidence, but 
some features, such as strong evidence of a dose- 
response relationship or evidence of a very large effect 
[11], increase our confidence. Importantly, this confi- 
dence differs by outcome such that, for example, we may 
have more confidence in the effects of an intervention on 
patient important benefits while having low confidence 
in its effects on harms. 
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Empirical evidence regarding this quality 
dimension 

To evaluate the current practice of including evidence in 

decision aids, we examined the Ottawa Decision Aid 
Inventory (http://decisionaid.ohri.ca/azinvent.php). Out 
of 257 decision aids included in the inventory, 134 pro- 
vided references to scientific evidence used, when they 
were last updated (n = 135), and whether they were 
available on the Internet (n = 134). In a random sample 
of those decision aids (n = 20), stratified to correct for 
differences between providers, ten aids used a high qual- 
ity systematic review/meta-analysis (AMSTAR score [12] 
ranging from 2 to 11 on a scale of 1-11), and five aids 
were based on practice guidelines (AGREE II score [13] 
ranging from 1 to 6 on a scale of 1-7). Two decision 
aids used data from multiple sources of original 
research, whereas two used only a narrative review or 
an expert's opinion or a single piece of original research 
as the sources for evidence. Four decision aids did not 
explicitly cite, hence we could not locate and evaluate, 
the evidence they used. One of the 20 aids explicitly sta- 
ted an expiration date and an update policy, whereas 
eight refer to a policy statement of the complete con- 
tents on the provider website (as part of a site notice on 
the website, but not as part of the aid itself). Three used 
the GRADE system in their presentation to clarify the 
quality of the evidence to the user. 

Discussion 

Linking the quality of a patient decision aid to the quality 
of the process by which developers comprehensively iden- 
tified, synthesized, and appraised up-to-date evidence is a 
key dimension represented in the IPDAS. Furthermore, 
the little empirical evidence available suggests much room 
for improvement. The risk with standards is to suggest 
that no uncertainties remain or that innovation has stalled. 
Neither is true with relation to this topic. In working 
through these issues, the authors identified the following 
emerging topics and research areas for discussion. The 
process by which these areas were identified and are high- 
lighted here does not pretend to be systematic or all- 
encompassing, but rather serves as a starting point for a 
community-wide conversation. 

How might developers decide what evidence used to 
Inform a decision might be pertinent to the patients who 
are the intended audience of the tool? 

The extent to which evidence is pertinent is subject to 
much judgment. Users may end up having low confi- 
dence in estimates of subgroup effects (e.g., subgroup 
effects that cannot be confirmed or are very imprecise), 
such that it is sometimes advisable to use estimates from 
the general population [14]. The challenge of applying 
evidence from somewhat different patients, interventions. 



or outcomes to the situation of interest falls under the 
general rubric of indirectness [10]. The degree with 
which these differences are likely to undermine the 
applicability of the evidence— that is, the extent of indir- 
ectness—reduces the confidence that the estimates of 
effect are correct, and this could be reflected in the deci- 
sion aid. Efforts to improve the volume and quality of 
comparative effectiveness research [15] may enhance the 
evidence base for decision aids, because this research 
requires direct comparisons that matter to clinical stake- 
holders - that is, measuring the effect of interventions on 
outcomes of importance to patients. 

How often should systematic reviews be updated? 

The frequency with which systematic reviews should be 
updated - and, by extension, the products derived from 
these reviews - is still subject to research. A comprehen- 
sive technical review commissioned by the Agency for 
Healthcare Research and Quality and published in 2007 
[16] found that the median time for the emergence of a 
signal that a review should be updated (e.g., substantial 
new evidence of effectiveness or harm, new alternatives, 
revelations about the nature of the old evidence) was 5.5 
years; yet about 25% of the reviews could benefit from 
updating within 2 years of publication. The authors were 
not able to identify predictors of more urgent review and 
suggested yearly surveillance of systematic reviews. 

How might we trust tailored estimates? 

Decision aids often need to present absolute risk esti- 
mates that relate to the individual characteristics of the 
patient of interest. There is considerable uncertainty 
about such baseline risks [17]. Prognostic calculators 
offer the opportunity to tailor such risk estimates. Given 
the importance of these estimates, decision aid develo- 
pers must report which calculator they are using and 
provide an assessment of its validity, a matter that often 
requires independent evaluation from the population 
from which the formula was derived, as well as compari- 
sons with competing risk estimators [18]. 

How might developers communicate their confidence in 
the estimates of effect? 

Communicating the confidence that decision aid develo- 
pers have in the estimates of the benefits and harms asso- 
ciated with the selected options would help decision aid 
users to make sense of the magnitude and trustworthiness 
of the estimates. The communication of this confidence in 
the evidence should be done in a way that is simple and 
understandable, yet remains precise, without adding cogni- 
tive burden on decision aid users. 

There is limited empirical work on how to communi- 
cate confidence in effect estimates to stakeholders in 
general, let alone decision aid users. A systematic review 
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of information in decision aids reported that very few 
decision aid developers had addressed or incorporated 
confidence in effect estimates in their tools [19]. The 
ways by which these developers were representing this 
concept varied greatly, from icons (i.e., bronze-gold 
medals), to verbal labels (i.e., high, moderate, low), and 
to numeric intervals, and these have yielded mixed 
results [19]. 

The GRADE approach for guidelines offers a simple way 
to report on confidence in effect estimates, using categori- 
cal labels— for example, "very low", "low", "moderate", and 
"high" confidence. [20,21]. An obvious advantage gained 
from using the same approach in guidelines and in deci- 
sion aids to rate confidence in effect estimates is that deci- 
sion aid developers could use the evidence supporting 
current high-quality guidelines as their source for up-to- 
date evidence. 

Will linking information to evidence-based practice 
guidelines foster decision aid uptake? 

A key issue with decision aids is their underuse in prac- 
tice, despite increasing evidence of their effectiveness 
and surging policy support in some regions. Thus, 
efforts to explicitly link their design and content to clin- 
ical policy and workflow may facilitate their adoption. 
This supports in part our approach of linking the devel- 
opment and content of decision aids to state-of-the-art 
approaches to the development and content of evi- 
dence-based practice guidelines. 

The "ideal" situation indicating the potential usefulness 
of a patient decision aid is one in which there is evidence 
producing high-confidence estimates linking options to 
outcomes, but the options are closely matched, and the 
choice of the "best" course of action will depend mostly 
on the patient's values and preferences. Guideline develo- 
pers following GRADE usually will offer a weak or condi- 
tional suggestion in those circumstances. A suggestion 
based on high-confidence evidence would require the 
incorporation of patient values and preferences for 
implementation, that is, would benefit from a decision 
aid. This linkage between guidelines and decision aids 
may affect the development and uptake of decision aids. 
Furthermore, implementation of the guidelines, for 
example, in quality improvement efforts, could then be 
linked to the implementation of decision aids for weak or 
conditional suggestions. 

What happens with patient decision aids that contradict 
extant practice guidelines? 

A key aspect worthy of surveillance is the fate of deci- 
sion aids that have been developed based on the best 
available evidence, but which disagree in their presenta- 
tion with extant guidelines and with the quality-of-care 
parameters derived from these guidelines. These 



discrepancies can appear because decision aids may use 
more recently updated summaries of evidence, or 
because the recommendations apply to a different con- 
text or resulted from panels with distorted values (e.g., 
due to pharmaceutical lobbying). In one example, this 
apparent divergence led to difficulties in the use of the 
decision aid and to its nonuse [22]. 

Will adherence to this dimension improve decision aid 
use in practice? 

Given that the uptake in practice of decision aids remains 
limited, there is no strong evidence that decision aids sup- 
ported by high quality and updated summaries of the body 
of evidence would be more likely to be taken up in prac- 
tice. It is plausible, however, that adherence to this domain 
may create a reputation for high credibility that could 
drive the uptake of decision aids. There is no a priori rea- 
son to believe that the quality of the process of evidence 
synthesis and its adaptation into evidence-based decision 
aids will lead to better decision processes and outcomes. 
Yet, basing their content on accurate rather than inaccu- 
rate estimates may lead to better outcomes. These central 
concerns, therefore, remain important research questions. 

Conclusions 

A key quality dimension for patient decision aids is how 
developers identified, summarized, and used research evi- 
dence to inform the content of these tools. Patients and 
clinicians have the expectation that these tools will sup- 
port evidence-based practice. Ethical, quality-of-care, and 
scientific arguments justify making this an IPDAS stan- 
dard. Empirical evidence suggests that there is much room 
for improvement in developers' adherence to this stan- 
dard. Applying the methodological developments of the 
GRADE working group to this task may help developers 
enhance their adherence to this domain and thus the qual- 
ity of their patient decision aids. It may also help position 
the development of decision aids in the same development 
stream as that of high quality clinical practice guidelines. 
Yet, much room exists for research and innovation into 
the processes that create patient decision aids based on 
comprehensive and up-to-date syntheses of all the perti- 
nent and critically appraised scientific evidence. 
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